Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5510 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9944. Min Act: -0.0002

Data Index: 473476 (C4 (Web Text))

Max Activating Token Index: 868

Text #1

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9476. Min Act: -0.0002

Data Index: 1158089 (C4 (Web Text))

Max Activating Token Index: 613

Text #2

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9492. Min Act: -0.0002

Data Index: 358303 (C4 (Web Text))

Max Activating Token Index: 652

Text #3

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9347. Min Act: -0.0002

Data Index: 1191764 (C4 (Web Text))

Max Activating Token Index: 153

Text #4

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9021. Min Act: -0.0002

Data Index: 878474 (C4 (Web Text))

Max Activating Token Index: 321

Text #5

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9143. Min Act: -0.0002

Data Index: 214351 (C4 (Web Text))

Max Activating Token Index: 713

Text #6

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.9027. Min Act: -0.0002

Data Index: 1293982 (C4 (Web Text))

Max Activating Token Index: 514

Text #7

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8952. Min Act: -0.0002

Data Index: 84989 (C4 (Web Text))

Max Activating Token Index: 282

Text #8

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8879. Min Act: -0.0002

Data Index: 982047 (C4 (Web Text))

Max Activating Token Index: 542

Text #9

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8835. Min Act: -0.0002

Data Index: 375883 (C4 (Web Text))

Max Activating Token Index: 711

Text #10

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8827. Min Act: -0.0002

Data Index: 303712 (C4 (Web Text))

Max Activating Token Index: 557

Text #11

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8799. Min Act: -0.0002

Data Index: 887310 (C4 (Web Text))

Max Activating Token Index: 727

Text #12

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8931. Min Act: -0.0002

Data Index: 167880 (C4 (Web Text))

Max Activating Token Index: 461

Text #13

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8831. Min Act: -0.0002

Data Index: 804402 (C4 (Web Text))

Max Activating Token Index: 908

Text #14

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8806. Min Act: -0.0002

Data Index: 668774 (C4 (Web Text))

Max Activating Token Index: 724

Text #15

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8704. Min Act: -0.0002

Data Index: 362635 (C4 (Web Text))

Max Activating Token Index: 602

Text #16

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8518. Min Act: -0.0002

Data Index: 1187166 (C4 (Web Text))

Max Activating Token Index: 933

Text #17

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8539. Min Act: -0.0002

Data Index: 137520 (C4 (Web Text))

Max Activating Token Index: 834

Text #18

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8412. Min Act: -0.0002

Data Index: 1147788 (C4 (Web Text))

Max Activating Token Index: 578

Text #19

Max Range: 0.9944. Min Range: -0.9944

Max Act: 0.8444. Min Act: -0.0002

Data Index: 760666 (C4 (Web Text))

Max Activating Token Index: 688