Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2097 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 11.2081. Min Range: -11.2081

Max Act: 11.2081. Min Act: -0.0000

Data Index: 629096 (C4 (Web Text))

Max Activating Token Index: 705

Text #1

Max Range: 11.2081. Min Range: -11.2081

Max Act: 10.2578. Min Act: -0.0000

Data Index: 148579 (C4 (Web Text))

Max Activating Token Index: 515

Text #2

Max Range: 11.2081. Min Range: -11.2081

Max Act: 9.5816. Min Act: -0.0000

Data Index: 1181963 (C4 (Web Text))

Max Activating Token Index: 570

Text #3

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.8873. Min Act: -0.0000

Data Index: 364222 (C4 (Web Text))

Max Activating Token Index: 626

Text #4

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.9996. Min Act: -0.0000

Data Index: 596427 (C4 (Web Text))

Max Activating Token Index: 559

Text #5

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.3847. Min Act: -0.0000

Data Index: 1279757 (C4 (Web Text))

Max Activating Token Index: 674

Text #6

Max Range: 11.2081. Min Range: -11.2081

Max Act: 9.0998. Min Act: -0.0000

Data Index: 650478 (C4 (Web Text))

Max Activating Token Index: 169

Text #7

Max Range: 11.2081. Min Range: -11.2081

Max Act: 6.9977. Min Act: -0.0000

Data Index: 1022966 (C4 (Web Text))

Max Activating Token Index: 797

Text #8

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.2463. Min Act: -0.0000

Data Index: 261826 (C4 (Web Text))

Max Activating Token Index: 680

Text #9

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.5643. Min Act: -0.0000

Data Index: 413804 (C4 (Web Text))

Max Activating Token Index: 473

Text #10

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.2705. Min Act: -0.0000

Data Index: 496086 (C4 (Web Text))

Max Activating Token Index: 481

Text #11

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.0130. Min Act: -0.0000

Data Index: 1328610 (C4 (Web Text))

Max Activating Token Index: 562

Text #12

Max Range: 11.2081. Min Range: -11.2081

Max Act: 9.2937. Min Act: -0.0000

Data Index: 840310 (C4 (Web Text))

Max Activating Token Index: 660

Text #13

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.3312. Min Act: -0.0000

Data Index: 957518 (C4 (Web Text))

Max Activating Token Index: 557

Text #14

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.0040. Min Act: -0.0000

Data Index: 325790 (C4 (Web Text))

Max Activating Token Index: 933

Text #15

Max Range: 11.2081. Min Range: -11.2081

Max Act: 8.0178. Min Act: -0.0000

Data Index: 1325090 (C4 (Web Text))

Max Activating Token Index: 703

Text #16

Max Range: 11.2081. Min Range: -11.2081

Max Act: 7.7266. Min Act: -0.0000

Data Index: 561992 (C4 (Web Text))

Max Activating Token Index: 128

Text #17

Max Range: 11.2081. Min Range: -11.2081

Max Act: 7.5442. Min Act: 0.0000

Data Index: 706837 (C4 (Web Text))

Max Activating Token Index: 696

Text #18

Max Range: 11.2081. Min Range: -11.2081

Max Act: 7.3471. Min Act: -0.0000

Data Index: 17246 (C4 (Web Text))

Max Activating Token Index: 383

Text #19

Max Range: 11.2081. Min Range: -11.2081

Max Act: 7.1132. Min Act: -0.0000

Data Index: 345957 (C4 (Web Text))

Max Activating Token Index: 339