Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1775 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.2856. Min Act: -0.0000

Data Index: 723479 (C4 (Web Text))

Max Activating Token Index: 71

Text #1

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.1655. Min Act: -0.0000

Data Index: 1148110 (C4 (Web Text))

Max Activating Token Index: 20

Text #2

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0696. Min Act: -0.0000

Data Index: 676861 (C4 (Web Text))

Max Activating Token Index: 121

Text #3

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0647. Min Act: -0.0000

Data Index: 10237 (C4 (Web Text))

Max Activating Token Index: 42

Text #4

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.1042. Min Act: -0.0000

Data Index: 1036086 (C4 (Web Text))

Max Activating Token Index: 11

Text #5

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.1927. Min Act: -0.0000

Data Index: 1107276 (C4 (Web Text))

Max Activating Token Index: 347

Text #6

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.1183. Min Act: -0.0000

Data Index: 835127 (C4 (Web Text))

Max Activating Token Index: 867

Text #7

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0357. Min Act: -0.0000

Data Index: 340463 (C4 (Web Text))

Max Activating Token Index: 371

Text #8

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0179. Min Act: -0.0000

Data Index: 2323 (C4 (Web Text))

Max Activating Token Index: 432

Text #9

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0793. Min Act: -0.0000

Data Index: 1168310 (C4 (Web Text))

Max Activating Token Index: 269

Text #10

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0709. Min Act: -0.0000

Data Index: 802174 (C4 (Web Text))

Max Activating Token Index: 84

Text #11

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0752. Min Act: -0.0000

Data Index: 20120 (C4 (Web Text))

Max Activating Token Index: 105

Text #12

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0571. Min Act: -0.0000

Data Index: 738388 (C4 (Web Text))

Max Activating Token Index: 309

Text #13

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.1116. Min Act: -0.0000

Data Index: 1156710 (C4 (Web Text))

Max Activating Token Index: 699

Text #14

Max Range: 1.2856. Min Range: -1.2856

Max Act: 1.0038. Min Act: -0.0000

Data Index: 797089 (C4 (Web Text))

Max Activating Token Index: 37

Text #15

Max Range: 1.2856. Min Range: -1.2856

Max Act: 0.9468. Min Act: -0.0000

Data Index: 655630 (C4 (Web Text))

Max Activating Token Index: 420

Text #16

Max Range: 1.2856. Min Range: -1.2856

Max Act: 0.9763. Min Act: -0.0000

Data Index: 274828 (C4 (Web Text))

Max Activating Token Index: 13

Text #17

Max Range: 1.2856. Min Range: -1.2856

Max Act: 0.9934. Min Act: -0.0000

Data Index: 1192113 (C4 (Web Text))

Max Activating Token Index: 111

Text #18

Max Range: 1.2856. Min Range: -1.2856

Max Act: 0.9906. Min Act: -0.0000

Data Index: 254907 (C4 (Web Text))

Max Activating Token Index: 28

Text #19

Max Range: 1.2856. Min Range: -1.2856

Max Act: 0.9027. Min Act: -0.0000

Data Index: 765255 (C4 (Web Text))

Max Activating Token Index: 39