[Sbcl-devel] Storing line numbers in code locations

[John Fremlin]

Dear SBCL,

First off, thanks for making SBCL. We recently did an informal survey of
the quality of code generated by Lisp compilers and SBCL was by far the
best.

However, one slightly annoying issue with SBCL is that it stores the
code locations for functions and other things in a way that does not
make it easy to get out the line number. IIUC, it stores the number of
toplevel forms through the file and then the number of the form inside
that toplevel form.

This means that to find a function in the source file, SLIME has to do
all sorts of shenanigans with reading in the file again (with an attempt
to replace the read-table) and then computing the line number.

Is there a philosophical objection to adding line/column numbers to code
locations? Is it technically difficult? Would a patch to do it be
considered, and if so should I remove the toplevel form counters at the
same time or leave them alone?

[Stas Boukarev]

Post by John Fremlin
Dear SBCL,
First off, thanks for making SBCL. We recently did an informal survey of
the quality of code generated by Lisp compilers and SBCL was by far the
best.
However, one slightly annoying issue with SBCL is that it stores the
code locations for functions and other things in a way that does not
make it easy to get out the line number. IIUC, it stores the number of
toplevel forms through the file and then the number of the form inside
that toplevel form.
This means that to find a function in the source file, SLIME has to do
all sorts of shenanigans with reading in the file again (with an attempt
to replace the read-table) and then computing the line number.
Is there a philosophical objection to adding line/column numbers to code
locations? Is it technically difficult? Would a patch to do it be
considered, and if so should I remove the toplevel form counters at the
same time or leave them alone?

This allows slime to find the right location of the form without
recompilation of the whole file, whenever you change it's layout, like
inserting new lines, forms, etc.

[Tobias C. Rittweiler]

Post by John Fremlin
Is there a philosophical objection to adding line/column numbers to code
locations? Is it technically difficult? Would a patch to do it be
considered, and if so should I remove the toplevel form counters at the
same time or leave them alone?

TLF numbers are more likely to be still accurate even when the user
edits the file and reloads just the new definitions interactively which
is the typical development model in Lisp.

And we do not only store the TLF number but also the path to the
actually offending form. Slime will highlight the actually offending
form on compiler notes, or if you use `v' in the debugger. That's *so*
much nicer than line numbers!

For example if in Java you have a line like that

sum = a[i] * b[j] + c[j-k];

And that causes an ArrayOutOfBounds exception, you'll get the line
number but that doesn't tell you much. In Slime, if you use `v' in the
debugger the offending (aref foo bar) will be highlighted. (Perhaps when
running under Eclipse the same is possible, mind you.)

So let's better not get rid of this kind of specifying locations. :-)

But see

http://jsnell.iki.fi/blog/archive/2007-12-19-pretty-sbcl-backtraces.html

which also shows you to get a line number with Swank. (And a comment
says that storing the line number explicitly may indeed be considered.)

-T.

[John Fremlin]

Post by Tobias C. Rittweiler

Post by John Fremlin
Is there a philosophical objection to adding line/column numbers to code
locations? Is it technically difficult? Would a patch to do it be
considered, and if so should I remove the toplevel form counters at the
same time or leave them alone?

TLF numbers are more likely to be still accurate even when the user
edits the file and reloads just the new definitions interactively which
is the typical development model in Lisp.
And we do not only store the TLF number but also the path to the
actually offending form. Slime will highlight the actually offending
form on compiler notes, or if you use `v' in the debugger. That's *so*
much nicer than line numbers!
For example if in Java you have a line like that
sum = a[i] * b[j] + c[j-k];
And that causes an ArrayOutOfBounds exception, you'll get the line
number but that doesn't tell you much. In Slime, if you use `v' in the
debugger the offending (aref foo bar) will be highlighted. (Perhaps when
running under Eclipse the same is possible, mind you.)
So let's better not get rid of this kind of specifying locations. :-)
But see
http://jsnell.iki.fi/blog/archive/2007-12-19-pretty-sbcl-backtraces.html
which also shows you to get a line number with Swank. (And a comment
says that storing the line number explicitly may indeed be
considered.)

Thanks.

In fact, the reason for bringing this up is that I am thinking about
backtraces and, in particular, unifying their format across a few
implementations.

I guess the consensus is that people like the TLF forms.

Would a patch to additionally add the line and column number be
considered?

The advantage is that it doesn't need the source file to be read in
again to give out the line number, which is a sort of standard for
specifying the location in a file.

[...]

[Leslie P. Polzer]

Post by John Fremlin
In fact, the reason for bringing this up is that I am thinking about
backtraces and, in particular, unifying their format across a few
implementations.
I guess the consensus is that people like the TLF forms.
Would a patch to additionally add the line and column number be
considered?

I am in favor of this, too. While TLF forms may have their merits
they are useless for me because I don't have a tool that will
walk the syntax tree and display the resulting form to me
(I could probably hack something with Swank, though).

With line numbers I could just point my editor at the offending
line.

Leslie

[John Fremlin]

Post by Leslie P. Polzer

Post by John Fremlin
In fact, the reason for bringing this up is that I am thinking about
backtraces and, in particular, unifying their format across a few
implementations.
I guess the consensus is that people like the TLF forms.
Would a patch to additionally add the line and column number be
considered?

I am in favor of this, too. While TLF forms may have their merits
they are useless for me because I don't have a tool that will
walk the syntax tree and display the resulting form to me
(I could probably hack something with Swank, though).

That is actually rather annoying because of the issue of replacing the
read-table, which swank has a good stab at.

Post by Leslie P. Polzer
With line numbers I could just point my editor at the offending
line.

I shall have a go at doing it in the next couple of weeks. If I fall
silent, then someone else please pick up the baton. ;-)

PS. Meddling with Clozure CL backtraces I find that it uses a single
"form number".

I guess that there is a massive hatred of the idea of line number in the
Lisp world.

[John Fremlin]

[...]

Post by John Fremlin

Post by Leslie P. Polzer

Post by John Fremlin
Would a patch to additionally add the line and column number be
considered?

I am in favor of this, too. While TLF forms may have their merits
they are useless for me because I don't have a tool that will
walk the syntax tree and display the resulting form to me

[...]

Post by John Fremlin
I shall have a go at doing it in the next couple of weeks. If I fall
silent, then someone else please pick up the baton. ;-)

Okay, here's a rough sketch of what I'm doing so that I can quickly be
told off for going about it the wrong way.

I want to make a sort of modified stream that keeps track of the line
and column number for reading in lisp files that will be
compiled. Normally one could use a generic stream (Gray stream). Is this
allowed at this stage in the compiler? Is there one already?


(Obviously not a patch to be applied.)


src/code/debug-int.lisp
defstruct code-location, add line and column number
+ (line-number (sb!c:read-var-integer blocks i))
+ (column-number (sb!c:read-var-integer blocks i))


diff --git a/src/compiler/debug-dump.lisp b/src/compiler/debug-dump.lisp
index 93dc016..6e458be 100644
--- a/src/compiler/debug-dump.lisp
+++ b/src/compiler/debug-dump.lisp
@@ -110,6 +110,9 @@
(write-var-integer (source-path-tlf-number path) *byte-buffer*))
(write-var-integer (source-path-form-number path) *byte-buffer*))

+ (write-var-integer (node-line-number node) *byte-buffer*)
+ (write-var-integer (node-column-number node) *byte-buffer*)
+

diff --git a/src/compiler/ir1tran-lambda.lisp b/src/compiler/ir1tran-lambda.lisp
index b107a41..7cd60c0 100644
--- a/src/compiler/ir1tran-lambda.lisp
+++ b/src/compiler/ir1tran-lambda.lisp
@@ -880,7 +880,9 @@
:%debug-name debug-name
:plist `(:ir1-environment
(,*lexenv*
- ,*current-path*))))
+ ,*current-path*
+ ,*current-line-number
+ ,*current-column-number))))
(min (or (position-if #'lambda-var-arg-info vars) (length vars))))
(aver-live-component *current-component*)
(push res (component-new-functionals *current-component*))
@@ -1198,7 +1200,7 @@
(defun optional-dispatch-entry-point-fun (dispatcher n)
(declare (type optional-dispatch dispatcher)
(type unsigned-byte n))
- (let* ((env (getf (optional-dispatch-plist dispatcher) :ir1-environment))
- (*lexenv* (first env))
- (*current-path* (second env)))
- (force (nth n (optional-dispatch-entry-points dispatcher)))))
+ (let ((env (getf (optional-dispatch-plist dispatcher) :ir1-environment)))
+ (destructuring-bind (*lexenv* *current-path* *current-line-number* *current-column-number*)
+ env
+ (force (nth n (optional-dispatch-entry-points dispatcher))))))
diff --git a/src/compiler/macros.lisp b/src/compiler/macros.lisp
index 7ec86bd..5ac9ae0 100644
--- a/src/compiler/macros.lisp
+++ b/src/compiler/macros.lisp
@@ -672,7 +672,9 @@
(declare (type node node) (type function fun))
(let ((*current-component* (node-component node))
(*lexenv* (node-lexenv node))
- (*current-path* (node-source-path node)))
+ (*current-path* (node-source-path node))
+ (*current-line-number* (node-line-number node))
+ (*current-column-number* (node-column-number node)))
(aver-live-component *current-component*)
(funcall fun)))

diff --git a/src/compiler/node.lisp b/src/compiler/node.lisp
index cb167fd..6e075b4 100644
--- a/src/compiler/node.lisp
+++ b/src/compiler/node.lisp
@@ -127,6 +127,8 @@
;; is the ordinal number (in this call to the compiler) of the truly
;; top level form containing the original source.
(source-path *current-path* :type list)
+ (line-number *current-line-number* :type index)
+ (column-number *current-column-number* :type index)
;; If this node is in a tail-recursive position, then this is set to
;; T. At the end of IR1 (in physical environment analysis) this is
;; computed for all nodes (after cleanup code has been emitted).




[...]

[Nikodemus Siivola]

Post by John Fremlin
I want to make a sort of modified stream that keeps track of the line
and column number for reading in lisp files that will be
compiled. Normally one could use a generic stream (Gray stream). Is this
allowed at this stage in the compiler? Is there one already?

No, you can't use a Gray steam -- no CLOS before warm init in the
build. You should be able to define your own new subclass/structure of
ANSI-STREAM, though, and wrap the normal file stream in that: M-.
concatenated-stream will take you to the right area.

Cheers,

-- Nikodemus Siivola