Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2078 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.2252. Min Act: -0.0000

Data Index: 650984 (C4 (Web Text))

Max Activating Token Index: 245

Text #1

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.2673. Min Act: -0.0000

Data Index: 562293 (C4 (Web Text))

Max Activating Token Index: 269

Text #2

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.0957. Min Act: -0.0000

Data Index: 224971 (C4 (Web Text))

Max Activating Token Index: 538

Text #3

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.1396. Min Act: -0.0000

Data Index: 414513 (C4 (Web Text))

Max Activating Token Index: 375

Text #4

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.1632. Min Act: -0.0000

Data Index: 653313 (C4 (Web Text))

Max Activating Token Index: 247

Text #5

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.0108. Min Act: -0.0000

Data Index: 446404 (C4 (Web Text))

Max Activating Token Index: 297

Text #6

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.9683. Min Act: -0.0000

Data Index: 887554 (C4 (Web Text))

Max Activating Token Index: 322

Text #7

Max Range: 1.2673. Min Range: -1.2673

Max Act: 1.0550. Min Act: -0.0000

Data Index: 365939 (C4 (Web Text))

Max Activating Token Index: 617

Text #8

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.9199. Min Act: -0.0000

Data Index: 399024 (C4 (Web Text))

Max Activating Token Index: 777

Text #9

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8559. Min Act: -0.0000

Data Index: 874038 (C4 (Web Text))

Max Activating Token Index: 182

Text #10

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8850. Min Act: -0.0000

Data Index: 602197 (C4 (Web Text))

Max Activating Token Index: 332

Text #11

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8926. Min Act: -0.0000

Data Index: 866372 (C4 (Web Text))

Max Activating Token Index: 519

Text #12

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.9515. Min Act: -0.0000

Data Index: 71011 (C4 (Web Text))

Max Activating Token Index: 535

Text #13

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.6905. Min Act: -0.0000

Data Index: 194409 (C4 (Web Text))

Max Activating Token Index: 593

Text #14

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.7994. Min Act: -0.0000

Data Index: 243532 (C4 (Web Text))

Max Activating Token Index: 159

Text #15

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8584. Min Act: -0.0000

Data Index: 113773 (C4 (Web Text))

Max Activating Token Index: 411

Text #16

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8159. Min Act: -0.0000

Data Index: 278586 (C4 (Web Text))

Max Activating Token Index: 62

Text #17

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8372. Min Act: -0.0000

Data Index: 711635 (C4 (Web Text))

Max Activating Token Index: 782

Text #18

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.7274. Min Act: -0.0000

Data Index: 196179 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 1.2673. Min Range: -1.2673

Max Act: 0.8202. Min Act: -0.0000

Data Index: 483169 (C4 (Web Text))

Max Activating Token Index: 717