Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 86 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.0932. Min Range: -6.0932

Max Act: 6.0563. Min Act: -0.0000

Data Index: 1155700 (C4 (Web Text))

Max Activating Token Index: 418

Text #1

Max Range: 6.0932. Min Range: -6.0932

Max Act: 6.0518. Min Act: -0.0000

Data Index: 810844 (C4 (Web Text))

Max Activating Token Index: 446

Text #2

Max Range: 6.0932. Min Range: -6.0932

Max Act: 5.7853. Min Act: -0.0000

Data Index: 956735 (C4 (Web Text))

Max Activating Token Index: 661

Text #3

Max Range: 6.0932. Min Range: -6.0932

Max Act: 6.0932. Min Act: -0.0000

Data Index: 1263104 (C4 (Web Text))

Max Activating Token Index: 1005

Text #4

Max Range: 6.0932. Min Range: -6.0932

Max Act: 5.6592. Min Act: -0.0000

Data Index: 796796 (C4 (Web Text))

Max Activating Token Index: 879

Text #5

Max Range: 6.0932. Min Range: -6.0932

Max Act: 5.9644. Min Act: -0.0000

Data Index: 959196 (C4 (Web Text))

Max Activating Token Index: 576

Text #6

Max Range: 6.0932. Min Range: -6.0932

Max Act: 5.6872. Min Act: -0.0000

Data Index: 409567 (C4 (Web Text))

Max Activating Token Index: 65

Text #7

Max Range: 6.0932. Min Range: -6.0932

Max Act: 5.6845. Min Act: -0.0000

Data Index: 307442 (C4 (Web Text))

Max Activating Token Index: 996

Text #8

Max Range: 6.0932. Min Range: -6.0932

Max Act: 4.7724. Min Act: -0.0000

Data Index: 234611 (C4 (Web Text))

Max Activating Token Index: 951

Text #9

Max Range: 6.0932. Min Range: -6.0932

Max Act: 4.4593. Min Act: -0.0000

Data Index: 1161377 (C4 (Web Text))

Max Activating Token Index: 480

Text #10

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.9174. Min Act: -0.0000

Data Index: 141932 (C4 (Web Text))

Max Activating Token Index: 609

Text #11

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.5578. Min Act: -0.0000

Data Index: 263119 (C4 (Web Text))

Max Activating Token Index: 215

Text #12

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.9284. Min Act: -0.0000

Data Index: 938708 (C4 (Web Text))

Max Activating Token Index: 49

Text #13

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.4489. Min Act: -0.0000

Data Index: 802133 (C4 (Web Text))

Max Activating Token Index: 991

Text #14

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.8019. Min Act: -0.0000

Data Index: 869087 (C4 (Web Text))

Max Activating Token Index: 779

Text #15

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.5334. Min Act: -0.0000

Data Index: 1077096 (C4 (Web Text))

Max Activating Token Index: 148

Text #16

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.5506. Min Act: -0.0000

Data Index: 16158 (C4 (Web Text))

Max Activating Token Index: 572

Text #17

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.4745. Min Act: -0.0000

Data Index: 777459 (C4 (Web Text))

Max Activating Token Index: 649

Text #18

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.1876. Min Act: -0.0000

Data Index: 1083225 (C4 (Web Text))

Max Activating Token Index: 278

Text #19

Max Range: 6.0932. Min Range: -6.0932

Max Act: 3.7413. Min Act: -0.0000

Data Index: 609531 (C4 (Web Text))

Max Activating Token Index: 888