In this lecture, we're going to create the token structures.
So please go to compiler dot H.
We're just going to go struct token and we're just going to go init type init flags and we're going
to have a union.
And I'm sure you all know what a union is.
But for those who don't know a union, everything inside there is shared, the memory is shared.
So if we have an infrastructure, both of those variables share the same memory.
It's important to mention that for those that don't know what a union is.
And inside our union, we're going to just create some variables.
So we're going to have char see val cons char as val.
Unsigned init enum.
Unsigned long l name and unsigned long long l name.
And then we're just going to go void any so that'll be a void pointer any.
So yeah, essentially our tokens value can be many different things.
It can be a char, it can be an integer, it can be a long it can be a pointer to a string.
So that's why we use a union here, right?
So we don't consume more memory than we need and we're guaranteed to only ever have one of these set
per value token.
So we're not going to be able to have a string token that's also an integer token, you know, so therefore
union is appropriate here.
And yeah, so during lexical analysis, if we came across a char, you know, something between these
quotes, then we would set the c vel.
We came across a string like something between these we would set the se val we come across a number,
we'd set the, the alum or the integer, right?
So that's how that works.
Our token will represent all of those possible values.
So now we're going to do pool whitespace and this basically signifies is there whitespace between the
token and the next token?
So I'm just going to put a comma there.
So I hope that explains it.
So essentially it's true if there's whitespace between the token and next token.
So Asterisk A would ensure that the white space Berlin is true for token A because there's white space
here, right?
You see that white space by that operator?
Okay.
So we're also going to have lunch between brackets.
Just like that.
So the between brackets variable will essentially point to a string that shows what was between brackets.
So for example, if we had something like Hello World, then every token inside of the of these brackets
would have between brackets pointing to the start of this expression, if that makes sense.
So if we did five plus ten plus 20, then four, five, four, ten and 420, they would all point to
the start of this buffer here.
Okay.
To the start of this bracket.
So this would say five plus ten plus 20 as a string for every token in here.
And you might wonder why we're doing that.
It's because of debugging when things go wrong, which they they might we need to have an efficient
way of being able to easily see what was between the brackets.
Okay, so that's what this is for.
So we also want to include std build H to resolve that white space sancta error.
So what else does a token need to know?
Well, we need to know the position in the file, of course.
So let's create a structure for that struck pos.
So what does the position contain?
What it contains?
A line number and a column.
And what else does a token contain?
A file name?
Okay.
So you might be wondering, why do we need a file name on a position?
Well, what are we doing here?
We're including studio.
We're including nested ball.
Right.
If our compiler goes to those files and gets tokens on them, wouldn't it be pretty wise to know which
file the token was on?
Was the token in this source file or the one we included?
That's why we have the file name, guys.
All right, let's create an enumeration here for the token types.
Okay.
Token type identifier.
Token type keyword.
Token type.
Operator.
Token type symbol.
Token type number.
Token type string.
Token type comment.
Token type new line.
Yeah.
So there are all the token types we need in our compiler guys.
So as you saw in the recent lecture about lexical analysis, you know, keywords, operator symbols,
numbers, strings, comment, new line, you know, they were all explained in that video.
These are all the ones we need.
And these will be set to the type, right?
So if we go through our source file and we find a number, then we'll create a token with type number.
And the, the, the enum will be set to whatever the number was or maybe the LX enum, depending on
the type of number by parsing flags will get that later.
There's really no need to worry about flags until we're actually starting to write the code.
So congratulations.
You now have the structure of the token and even line positioning.
In the next few lectures we'll start creating Alexa and using these token structures we have created.