Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 592 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1094. Min Range: -2.1094

Max Act: 2.1094. Min Act: -0.0000

Data Index: 567215 (C4 (Web Text))

Max Activating Token Index: 953

Text #1

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.8731. Min Act: -0.0000

Data Index: 1064166 (C4 (Web Text))

Max Activating Token Index: 625

Text #2

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.8211. Min Act: -0.0000

Data Index: 908107 (C4 (Web Text))

Max Activating Token Index: 627

Text #3

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.7121. Min Act: -0.0000

Data Index: 397850 (C4 (Web Text))

Max Activating Token Index: 497

Text #4

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5676. Min Act: -0.0000

Data Index: 858919 (C4 (Web Text))

Max Activating Token Index: 420

Text #5

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.7493. Min Act: -0.0000

Data Index: 74836 (C4 (Web Text))

Max Activating Token Index: 557

Text #6

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.6362. Min Act: -0.0000

Data Index: 565088 (C4 (Web Text))

Max Activating Token Index: 714

Text #7

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5565. Min Act: -0.0000

Data Index: 39598 (C4 (Web Text))

Max Activating Token Index: 368

Text #8

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.6021. Min Act: -0.0000

Data Index: 1306088 (C4 (Web Text))

Max Activating Token Index: 911

Text #9

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5968. Min Act: -0.0000

Data Index: 177837 (C4 (Web Text))

Max Activating Token Index: 933

Text #10

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.4781. Min Act: -0.0000

Data Index: 695412 (C4 (Web Text))

Max Activating Token Index: 416

Text #11

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5584. Min Act: -0.0000

Data Index: 1129460 (C4 (Web Text))

Max Activating Token Index: 225

Text #12

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5376. Min Act: -0.0000

Data Index: 1113645 (C4 (Web Text))

Max Activating Token Index: 366

Text #13

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5862. Min Act: -0.0000

Data Index: 263195 (C4 (Web Text))

Max Activating Token Index: 696

Text #14

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.5585. Min Act: -0.0000

Data Index: 1233905 (C4 (Web Text))

Max Activating Token Index: 406

Text #15

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.4759. Min Act: -0.0000

Data Index: 752195 (C4 (Web Text))

Max Activating Token Index: 626

Text #16

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.4943. Min Act: -0.0000

Data Index: 169885 (C4 (Web Text))

Max Activating Token Index: 268

Text #17

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.4783. Min Act: -0.0000

Data Index: 974527 (C4 (Web Text))

Max Activating Token Index: 408

Text #18

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.3965. Min Act: -0.0000

Data Index: 256715 (C4 (Web Text))

Max Activating Token Index: 806

Text #19

Max Range: 2.1094. Min Range: -2.1094

Max Act: 1.4699. Min Act: -0.0000

Data Index: 374201 (C4 (Web Text))

Max Activating Token Index: 127