Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 61 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1664. Min Range: -1.1664

Max Act: 1.1664. Min Act: -0.0000

Data Index: 194007 (C4 (Web Text))

Max Activating Token Index: 226

Text #1

Max Range: 1.1664. Min Range: -1.1664

Max Act: 1.0465. Min Act: -0.0000

Data Index: 870348 (C4 (Web Text))

Max Activating Token Index: 767

Text #2

Max Range: 1.1664. Min Range: -1.1664

Max Act: 1.0402. Min Act: -0.0000

Data Index: 892490 (C4 (Web Text))

Max Activating Token Index: 919

Text #3

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.9700. Min Act: -0.0000

Data Index: 126216 (C4 (Web Text))

Max Activating Token Index: 843

Text #4

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.9184. Min Act: -0.0000

Data Index: 912877 (C4 (Web Text))

Max Activating Token Index: 690

Text #5

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.9391. Min Act: -0.0000

Data Index: 65805 (C4 (Web Text))

Max Activating Token Index: 978

Text #6

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.9087. Min Act: -0.0000

Data Index: 568813 (C4 (Web Text))

Max Activating Token Index: 848

Text #7

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8784. Min Act: -0.0000

Data Index: 469267 (C4 (Web Text))

Max Activating Token Index: 770

Text #8

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.9185. Min Act: -0.0000

Data Index: 522216 (C4 (Web Text))

Max Activating Token Index: 783

Text #9

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8820. Min Act: -0.0000

Data Index: 643488 (C4 (Web Text))

Max Activating Token Index: 796

Text #10

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8421. Min Act: -0.0000

Data Index: 1333043 (C4 (Web Text))

Max Activating Token Index: 243

Text #11

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8162. Min Act: -0.0000

Data Index: 854154 (C4 (Web Text))

Max Activating Token Index: 382

Text #12

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8138. Min Act: -0.0000

Data Index: 14603 (C4 (Web Text))

Max Activating Token Index: 714

Text #13

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8618. Min Act: -0.0000

Data Index: 14827 (C4 (Web Text))

Max Activating Token Index: 103

Text #14

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8123. Min Act: -0.0000

Data Index: 1276477 (C4 (Web Text))

Max Activating Token Index: 993

Text #15

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.8228. Min Act: -0.0000

Data Index: 1049947 (C4 (Web Text))

Max Activating Token Index: 705

Text #16

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.7787. Min Act: -0.0000

Data Index: 937688 (C4 (Web Text))

Max Activating Token Index: 979

Text #17

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.7670. Min Act: -0.0000

Data Index: 387840 (C4 (Web Text))

Max Activating Token Index: 533

Text #18

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.7289. Min Act: -0.0000

Data Index: 1151964 (C4 (Web Text))

Max Activating Token Index: 105

Text #19

Max Range: 1.1664. Min Range: -1.1664

Max Act: 0.7657. Min Act: -0.0000

Data Index: 54858 (C4 (Web Text))

Max Activating Token Index: 158