Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1020 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.6471. Min Act: -0.0002

Data Index: 1321613 (C4 (Web Text))

Max Activating Token Index: 234

Text #1

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.5798. Min Act: -0.0002

Data Index: 445581 (C4 (Web Text))

Max Activating Token Index: 446

Text #2

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.5098. Min Act: -0.0002

Data Index: 591907 (C4 (Web Text))

Max Activating Token Index: 381

Text #3

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.5005. Min Act: -0.0002

Data Index: 555904 (C4 (Web Text))

Max Activating Token Index: 7

Text #4

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4695. Min Act: -0.0002

Data Index: 916235 (C4 (Web Text))

Max Activating Token Index: 36

Text #5

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4995. Min Act: -0.0002

Data Index: 922414 (C4 (Web Text))

Max Activating Token Index: 133

Text #6

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4899. Min Act: -0.0002

Data Index: 509154 (C4 (Web Text))

Max Activating Token Index: 170

Text #7

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4717. Min Act: -0.0002

Data Index: 906524 (C4 (Web Text))

Max Activating Token Index: 57

Text #8

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4465. Min Act: -0.0002

Data Index: 955846 (C4 (Web Text))

Max Activating Token Index: 126

Text #9

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4453. Min Act: -0.0002

Data Index: 760050 (C4 (Web Text))

Max Activating Token Index: 26

Text #10

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4450. Min Act: -0.0002

Data Index: 207404 (C4 (Web Text))

Max Activating Token Index: 386

Text #11

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4830. Min Act: -0.0002

Data Index: 1119604 (C4 (Web Text))

Max Activating Token Index: 167

Text #12

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4694. Min Act: -0.0002

Data Index: 1193148 (C4 (Web Text))

Max Activating Token Index: 581

Text #13

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4716. Min Act: -0.0002

Data Index: 237835 (C4 (Web Text))

Max Activating Token Index: 323

Text #14

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4421. Min Act: -0.0002

Data Index: 929911 (C4 (Web Text))

Max Activating Token Index: 788

Text #15

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4304. Min Act: -0.0002

Data Index: 399320 (C4 (Web Text))

Max Activating Token Index: 217

Text #16

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4420. Min Act: -0.0002

Data Index: 201670 (C4 (Web Text))

Max Activating Token Index: 604

Text #17

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4219. Min Act: -0.0002

Data Index: 763298 (C4 (Web Text))

Max Activating Token Index: 226

Text #18

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.4122. Min Act: -0.0002

Data Index: 1203200 (C4 (Web Text))

Max Activating Token Index: 642

Text #19

Max Range: 1.6471. Min Range: -1.6471

Max Act: 1.3927. Min Act: -0.0002

Data Index: 1159642 (C4 (Web Text))

Max Activating Token Index: 117