Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1744 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.4927. Min Act: -0.0000

Data Index: 1289739 (C4 (Web Text))

Max Activating Token Index: 778

Text #1

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.5990. Min Act: -0.0000

Data Index: 883203 (C4 (Web Text))

Max Activating Token Index: 980

Text #2

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.3688. Min Act: -0.0000

Data Index: 983003 (C4 (Web Text))

Max Activating Token Index: 414

Text #3

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.3745. Min Act: -0.0000

Data Index: 1073575 (C4 (Web Text))

Max Activating Token Index: 381

Text #4

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.3565. Min Act: -0.0000

Data Index: 1141392 (C4 (Web Text))

Max Activating Token Index: 736

Text #5

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.3233. Min Act: -0.0000

Data Index: 1178824 (C4 (Web Text))

Max Activating Token Index: 588

Text #6

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.3452. Min Act: -0.0000

Data Index: 13070 (C4 (Web Text))

Max Activating Token Index: 271

Text #7

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.2589. Min Act: -0.0000

Data Index: 980662 (C4 (Web Text))

Max Activating Token Index: 463

Text #8

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.1918. Min Act: -0.0000

Data Index: 810419 (C4 (Web Text))

Max Activating Token Index: 434

Text #9

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.1638. Min Act: -0.0000

Data Index: 493144 (C4 (Web Text))

Max Activating Token Index: 755

Text #10

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.1257. Min Act: -0.0000

Data Index: 965922 (C4 (Web Text))

Max Activating Token Index: 829

Text #11

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.0562. Min Act: -0.0000

Data Index: 81236 (C4 (Web Text))

Max Activating Token Index: 336

Text #12

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.9955. Min Act: -0.0000

Data Index: 876159 (C4 (Web Text))

Max Activating Token Index: 216

Text #13

Max Range: 1.5990. Min Range: -1.5990

Max Act: 1.0218. Min Act: -0.0000

Data Index: 980125 (C4 (Web Text))

Max Activating Token Index: 113

Text #14

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.9690. Min Act: -0.0000

Data Index: 1289943 (C4 (Web Text))

Max Activating Token Index: 180

Text #15

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.9457. Min Act: -0.0000

Data Index: 185712 (C4 (Web Text))

Max Activating Token Index: 19

Text #16

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.8949. Min Act: -0.0000

Data Index: 1056609 (C4 (Web Text))

Max Activating Token Index: 390

Text #17

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.8807. Min Act: -0.0000

Data Index: 888699 (C4 (Web Text))

Max Activating Token Index: 375

Text #18

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.8767. Min Act: -0.0000

Data Index: 417676 (C4 (Web Text))

Max Activating Token Index: 78

Text #19

Max Range: 1.5990. Min Range: -1.5990

Max Act: 0.8461. Min Act: -0.0000

Data Index: 955312 (C4 (Web Text))

Max Activating Token Index: 522