Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5033 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0904. Min Range: -1.0904

Max Act: 1.0904. Min Act: -0.0000

Data Index: 1097983 (C4 (Web Text))

Max Activating Token Index: 351

Text #1

Max Range: 1.0904. Min Range: -1.0904

Max Act: 1.0300. Min Act: -0.0000

Data Index: 1019692 (C4 (Web Text))

Max Activating Token Index: 404

Text #2

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.9968. Min Act: -0.0000

Data Index: 982680 (C4 (Web Text))

Max Activating Token Index: 417

Text #3

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.9029. Min Act: -0.0000

Data Index: 967037 (C4 (Web Text))

Max Activating Token Index: 750

Text #4

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.9108. Min Act: -0.0000

Data Index: 812016 (C4 (Web Text))

Max Activating Token Index: 18

Text #5

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.8702. Min Act: -0.0000

Data Index: 188775 (C4 (Web Text))

Max Activating Token Index: 1021

Text #6

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.8559. Min Act: -0.0000

Data Index: 762137 (C4 (Web Text))

Max Activating Token Index: 515

Text #7

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7746. Min Act: -0.0000

Data Index: 784992 (C4 (Web Text))

Max Activating Token Index: 203

Text #8

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7719. Min Act: -0.0000

Data Index: 1095966 (C4 (Web Text))

Max Activating Token Index: 832

Text #9

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7952. Min Act: -0.0000

Data Index: 870929 (C4 (Web Text))

Max Activating Token Index: 72

Text #10

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.8161. Min Act: -0.0000

Data Index: 330 (C4 (Web Text))

Max Activating Token Index: 369

Text #11

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7890. Min Act: -0.0000

Data Index: 182853 (C4 (Web Text))

Max Activating Token Index: 357

Text #12

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7992. Min Act: -0.0000

Data Index: 767473 (C4 (Web Text))

Max Activating Token Index: 641

Text #13

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7259. Min Act: -0.0000

Data Index: 785201 (C4 (Web Text))

Max Activating Token Index: 1013

Text #14

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7694. Min Act: -0.0000

Data Index: 499317 (C4 (Web Text))

Max Activating Token Index: 643

Text #15

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7435. Min Act: -0.0000

Data Index: 438267 (C4 (Web Text))

Max Activating Token Index: 691

Text #16

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7665. Min Act: -0.0000

Data Index: 285909 (C4 (Web Text))

Max Activating Token Index: 269

Text #17

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.7233. Min Act: -0.0000

Data Index: 1164624 (C4 (Web Text))

Max Activating Token Index: 549

Text #18

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.6902. Min Act: -0.0000

Data Index: 816804 (C4 (Web Text))

Max Activating Token Index: 906

Text #19

Max Range: 1.0904. Min Range: -1.0904

Max Act: 0.6975. Min Act: -0.0000

Data Index: 720560 (C4 (Web Text))

Max Activating Token Index: 819