Welcome back, everybody.
We're now going to start actually writing Alexa functionality.
So in the past couple of lectures, as you know, we've started creating some of the template functions,
getting the skeleton going for the Lexa.
Now we're going to start filling it.
So just open the X function here in Lexa Darcy and we're going to begin writing some code.
So we're going to start by going process.
Current expression count equals zero process parenthesis buffer equals null process postop file name
equals process compiler c file dot absolute path.
And just by here, we're going to go let's process equals process and we're going to need to define
Lack's process at the top here as well.
So just go lacks process.
Lacks process.
We're going to make that static.
So we now just need to make a read.
Next token functions just by here we're going to go struct token read next token.
So this function will be responsible for reading the next token from the file.
So essentially it will read like a character from the file and it will convert it to a token.
So we're just going to go struct token token equals null and we're just going to return the token here.
Okay.
That will be it for that function for now.
We'll fill it in later.
So now back to our function.
We're going to say struct token token equals read next token while token token equals read next token.
So this is basically going to read all the tokens from the input file from our C file here.
It's going to read all the tokens and every time we call read next token, it's going to read the next
token from the file.
So if it goes in here and it peeks and it sees an integer, it's going to read that whole number and
it's going to create an integer token.
Well, a number token, I should say.
So token type number.
You see, it'll create one of those and then it'll return it.
Okay.
So what are we going to do with this token?
Because we're not doing anything with the token right now.
Well, we're going to push it to our token vector.
So just go vector push.
And we're going to need to include the vector guys.
Factor push process token vec i was just going to pass in the token just like that.
And that will push the token to our token vector.
And rather than returning zero, we're just going to return lexical analysis all OC just like that.
So that's it for the next function now.
So we now need to make a peak C and a push C function.
So just appear do static char peak C.
Now, what's this going to do?
It's going to peak ahead in our file.
So how do we do that?
Well, we access our process, obviously, our function picture OC, and that's how that will work.
And this peak char will link.
To our peak cha function because you see the compiler lacks functions here that we set up.
We create a lexical analysis process that points to these functions.
So then when we then do process, function, peak cha.
It actually accesses compile a process picture.
So it will call this function, right, because we've abstracted it out.
So that's what's happening there.
And we now need to make one for push char.
So we're going to go static void, push, see.
And that's going to take a shower.
I'm just going to say lax process function, don't push cha.
We're going to put the legs process and the character and there we go.
So that's for peak and push, see?
Okay.
So back to read.
Next token, we're just going to go cha c equals pixie and now we're going to go switch C.
And here will be all our cases.
So if here we detect a numeric case, ask zero to ask zero to ASCII nine, then we'll handle a number
token, for example.
So the first thing we're going to do is end a file EAF.
We see an EAF.
We are done.
So we just break.
We have finished lexical analysis on the file, end of file EOF.
Okay.
So if we have a default, then there's an error.
Default basically means we've we've come across a character that we don't know what it is.
So for example, our compiler doesn't support Unicode.
So if someone put a Unicode symbol, this this default might be activated.
So we're just going to go compiler error and we don't have this function yet.
We're going to create it lacks process, compiler, unexpected token, just like that.
So we're now going to implement the compiler error function that we just wrote here.
So just go to compiler dot C and just by here we're going to go VOI compiler error struct compile process,
compiler const char message and then we're going to do the three dots because this is going to take
an infinite amount of arguments, right?
Little like printf.
So it's just going to include SDA, arg at the top here.
And just by here, we're going to create a VAR list and we're going to call that ARGs and we're going
to go var start args.
Message vrf print f std error message args and now we're going to go var and args and then we're going
to go f print f std error on line.
Make sure you put a space there as well, guys.
Percent a column, percent I n file percent se and when to do a new line there?
So compiler pos dot line compiler pascal compiler pos start file name just like that.
Finally, we're going to exit with a minus one in the event that this compile error happens.
And we're going to have to include SD Lib for that as well, guys for the exit function.
Okay.
Nice.
So this allows someone to call call our compiler error function as if they were calling printf.
They can provide they can provide a message, a message format.
And, you know, basically the percent I and stuff that we've done for this print, if they can provide
all of that here in our compiler error function.
So what we'll do is we will output our message that they provided straight to as to the error, but
only when we first parsed the arguments they provided a little like the ones we provided here to print
F right.
And then finally we output to the error on line and the line number on column and the column number
and the file name.
So that provides a nice compiler error message.
And just down here, we're going to say the same thing.
We're going to copy it and we're going to change it to warning and we're just going to remove the exit.
So that's a compiler warning.
Okay.
So that will just allow us to terminate the compiler at various parts of the compilation process when
things go wrong, a little like GC might tell you this variable does not is not been declared or something
like that.
We can use these functions to provide output to the user.
Right.
Just copy and paste these and we're going to put them in the header file just by here.
Okay.
Just like that.
Nice.
So now we can go back to the Lexa.
So we scroll down, we can see that on line 26 compiler now works.
Okay guys.
So let's just go to compiler because we need to create a numeric case.
So just scroll all the way to the top and we're just going to basically go, so now we're just going
to go hash tag define numeric case.
Case zero.
Case one.
Case two.
So essentially, guys, this will be inserted into a switch statement.
Okay.
So this is our numeric case all the way down to from 0 to 9 and then we're then going to go.
Case three.
Case four.
Case five
six.
Case seven.
Case a,
case nine.
And that is it.
And get rid of that slash there for case nine.
So if we now go back to dot C and we're just going to go here and we're just going to do numeric case,
right?
So this will basically paste all of those cases.
All of those cases will be pasted into this switch.
Case zero, case one, case two, and so on.
On, guys.
You can see it's complained about this code on here.
Just go back to the H, remove that final colon on case nine.
Now, if we go back to like six C, we can see new case now resolves to case or case one, case to case
three and so on.
So what do we do if it's a numeric case?
Well, we create a number token, obviously token equals token make number
OC and now we're going to create that function.
So we're going to say struct.
Token token make number.
Okay.
So what types of numbers are there?
There's hexadecimal numbers, there's decimal numbers, there's binary numbers.
We need to know which one we're working with.
So we're not just going to parse normal decimals here.
You know, normal decimals are not just going to be parsed.
We're going to parse hexadecimal and also binary.
So all of that is going to be done for this token make number.
So all we're going to do is we're going to do struct token token, make number for value and then we're
going to say an unsigned long number just like that.
And now if we go back to token number, we're going to say return token, make number for value, read
number, and we're going to have to make that function.
We'll do that now.
So just up here, we're going to say const char, read number, string, and we're going to say const
char num equals null.
And we're just going to go stroke buffer buffer equals buffer create.
And we need to include our buffer guys to just scroll up.
We're going to include helpers buffer h and then we're just going to say char C equals pixie.
And now we need to create special macro function.
This this macro function is going to be amazing, right?
Because we're going to use it in multiple places in the Laksa.
It's going to make our life so much easier.
So just go here and we're going to go hash tag, define Lex, get C, F, and that's going to take three
arguments, a buffer, a character and an expression.
So we're just going to go for C equals peek C and we're just going to go expression C equals PQ, C
OC.
And remember to use these slashes here so that it signifies we're making a new line on a macro definition
and we're just going to go.
Buffer, right.
Buffer C Okay.
And we're just going to do next.
C We'll create that in a moment, just like that.
So what does this allow us to do, which allows us to do things like that, like this, right?
Let's get C.F. Press and our buffer pass in our current char and then we can pass an expression.
So 50 above 90.
So 50 is above 90.
Then we'll write this character to the buffer.
That's basically what it does, right?
And it saves us having to use this four or five line for loop in multiple parts of our program because
we can pass the expression right in on what's acceptable for it to be added to the buffer.
Okay.
So that's what that function does.
So we now need to create the next C function.
So just do a static char next C and we're going to say char C equals lacks process function.
Next char lacks process.
And then we're going to say let's process dot pass.
Column plus equals one.
If C is equal to a new line character then lacks process.
Pascaline plus equals one lacks process, Pascal equals one.
And then we just return C So that just reads the next character from the, uh, from the file, right?
Which again, I showed you earlier, we set up the lacks process functions.
So that will call next char, which will get one character from the input file.
Okay.
So that's what that's about.
Okay.
So let's just scroll back down to our read number string.
So just by here, back at read number string, we're going to say let's get C, F, buffer, C and our
expression will be C above or equal as zero and C below equal ASCII nine.
So that's the algorithm for characters, right?
Is it above or equal to an ASCII zero and is it below or equal to an ASCII nine?
So that means if we pass something like that, that will be parsed and this number will be put into
this buffer as a ASCII string as seen here.
Okay.
So what do all strings have?
They have no terminators.
So let's just write an all terminator.
And now we're going to return the actual pointer to the actual buffer data like that.
So this buffer is really handy, right?
Because we can we can write to it as much as we want.
And what when it goes to too far into memory, it will reallocate itself, extend in the memory for
that string or for whatever data we're writing to the buffer.
Right.
So you can write to this as much as you want.
It will just keep re extending as you reach the boundaries.
So we're now going to make the read number.
So we're going to go unsigned long, long read number, constant char string will SW we'll call it read
number string
return a tol s.
So that should read the number and convert it to a long, long press.
So, so it passes in that in the test dot C file, the one, the one we're reading that will first be
put into the buffer and then convert it to a long, long integer.
So we're now going to create a token create function.
So just up here, we're going to say struct token, token, create struct token, underscore token.
I just need to scroll to the top here and we're going to create a static token here called the temp
token that'll be used for temporary measures when we're doing things that are only temporary.
So backs token create.
So we're going to do mem copy.
And we also need to include string hedge at the top here guys as well.
Just put it bye bye there.
That should be okay.
And that's going to copy into the temp token we created the token that's provided and the size of struct
token OC and we're then going to go temp token dot pass equals lax file position.
And I just want us to go to compile a dot hedge scroll down to token and I just need to add the position
in there like that.
OC Because we want to know where in the file a token is.
So by storing the position we'll know the exact line and column.
This token is on for the data that we read and convert it into a token.
So for example, if we had something like
48372871, then this position will be line one, column one for this 4837 token OC.
So go back to Lexa dot C and we're going to create the Lacks file position function which will return
the current position that the Lexa is at.
So this is very simple.
Just scroll up and we're going to say just by here.
Static struct pass.
Lacks file position return lacks process pass.
Just like that.
So just scroll back down to token create and we're just going to do return ampersand temp token, just
like that.
And now just scroll down to token, make number for value.
And we're just going to do return token.
Create ampersand struct token.
Dot type equals token type number de ll num equals number.
So that should be it.
We should now be at a read numbers.
So what we're going to do is we're going to make clean and make build this.
So open up your terminal and we're going to say make clean, make.
And we're just going to try and call main and we see unexpected token.
So that's fine.
Go to test.
See, we're going to remove our main function and we're just going to write an integer.
Five eight, three seven.
We can see that that worked.
Make another integer to 837 can see it says unexpected token.
That's probably because of the space.
Okay, so let's handle the actual white space to just go back to Alexa Dot, see?
We're going to scroll all the way to the top here.
And we're just going to say, let's send it here.
Static struct token handle whitespace and we're going to say struct token, less token equals lexa less
token.
Now, if we have a last token, then we're going to say last token whitespace equals true and then we're
going to do next, see.
And I know we haven't made the Lexa less token function yet.
Don't worry.
We'll do that in a minute and then we're then going to return read next token.
And I want to put this somewhere a bit more appropriate.
So just copy and paste it.
We're going to put it above number string.
So just by here we're going to say static struct token, lexa less token.
And this will return the the last token for us, right?
So we're just going to say return vector back on no legs process token vec now return the last token
that was pushed to the Lexa output to scroll down to here and we're going to just handle that.
So if it's a space or it's a tab, we don't care about them and we're going to ignore them.
So we're going to say handle white space.
We don't care about white space.
Ignore them.
Nice.
So back to the terminal.
We're going to try and make that again.
And you can see it's complaining.
That's because our read next token function is below the handle white space function.
So let's just copy and paste the actual prototype for read next token, go to the very top and we're
just going to paste it in just by that so that it gets seen before anything else do make again and we
can see that now works.
If you now run main, you can see it says everything compile fine.
So yeah we've created Alexa that's able to parse numbers
and I know I did say that we would be handling hexadecimal and binary in these functions earlier on
in this video.
We will handle that, but not in this lecture.
We'll get to that because it's a bit more complicated because it requires converting between the numbers
and that's a bit more complex.
So it won't be done in this in this lecture because this lecture is your first token.
So yeah, we will we will parse hexadecimal numbers such as this, but not in this lecture.
Just normal numbers for this lecture.
Okay, there's another thing I forgot to do.
Just head back to the Lexa, and we need to do token equals, handle whitespace.
We have to do that.
Because if you look here, only if there's a token, we continue.
So if we return null then then it'll end after it's read the first number.
Yeah.
So that's why handle whitespace will handle the whitespace and then return the next token.
Okay.
So let's now test this.
Go back to the terminal.
We're going to go make clean, make domain and everything, compile, fine, brilliant.
So let's now attach the GDB debugger to test our program.
So we're going to go GDB for main, we're just going to break on compile file and we're going to type
run.
And I would say it's broken on compile file function, we're going to type next and now we can just
press enter because that will run the the last command we used, which in this case was next.
So yeah, we're going to run the lexical analysis process there.
We just did it and now we're going to go print, asterisk, lex process and that'll basically print
out the entire Lex process structure to us.
If we just did Lex process, then it'll only print the pointer.
Yeah.
Which is why we have to use the asterisk to actually go in there so we can see the token vector set
correctly.
So we're just going to go lex process token vec and we're going to do a vector count.
We're going to call vector count on this token vector and we can see it gives us five, which is correct
because we have five tokens, five number tokens, one, two, three, four, five.
So that's correct.
So let's actually output the tokens.
Okay.
So what we're going to do is we're going to say print asterisk vector at lex process token VEC zero.
So that'll give us that will give us index zero.
And we also need to cast it to a struct token press enter and there we go.
There's our first token we can see is of type four which is number and we can see there is our number
in our union.
So you can see lx num 5837 and like I said, unions and C share the same memory, which is why you see
three of them here.
Yeah.
So allow us to access it as an integer or as a long or as a void pointer.
We can also see white space is true, signifying there's a white space between these numbers.
Okay, so that's all good.
Let's try index one.
We see 2837 index to three 6 to 7 index 31028 and index 44937.
So congratulations, you just created your first token in your Alexa.
So this is what lexical analysis is about.
No longer does the compiler have to worry about.
Oh, is that a number?
Is that a five?
Is an eight?
There are three.
It doesn't have to worry about any of that.
Lexical nasties worries about that.
And then for the rest of the compile process, we only have to worry about tokens.
We know that these are number tokens because of their type and we see their value and their value is
even an integer.
So we've actually converted it during lexical analysis time so we don't even have to worry about converting
it to an integer.
Okay, it's all nice and there for us to access.
So congratulations on creating your first token and in the next couple lectures we're going to continue
with Alexa and finish lexical analysis.
Thanks for watching.