Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 370 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.2897. Min Act: -0.0000

Data Index: 205851 (C4 (Web Text))

Max Activating Token Index: 518

Text #1

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.2766. Min Act: -0.0000

Data Index: 355254 (C4 (Web Text))

Max Activating Token Index: 484

Text #2

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.2030. Min Act: -0.0000

Data Index: 448536 (C4 (Web Text))

Max Activating Token Index: 448

Text #3

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.1899. Min Act: -0.0000

Data Index: 305660 (C4 (Web Text))

Max Activating Token Index: 69

Text #4

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.1114. Min Act: -0.0000

Data Index: 828229 (C4 (Web Text))

Max Activating Token Index: 546

Text #5

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.1083. Min Act: -0.0000

Data Index: 501961 (C4 (Web Text))

Max Activating Token Index: 177

Text #6

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.1079. Min Act: -0.0000

Data Index: 505750 (C4 (Web Text))

Max Activating Token Index: 50

Text #7

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0630. Min Act: -0.0000

Data Index: 567047 (C4 (Web Text))

Max Activating Token Index: 139

Text #8

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0546. Min Act: -0.0000

Data Index: 343250 (C4 (Web Text))

Max Activating Token Index: 676

Text #9

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0797. Min Act: -0.0000

Data Index: 1218032 (C4 (Web Text))

Max Activating Token Index: 926

Text #10

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0710. Min Act: -0.0000

Data Index: 938092 (C4 (Web Text))

Max Activating Token Index: 463

Text #11

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.1015. Min Act: -0.0000

Data Index: 845374 (C4 (Web Text))

Max Activating Token Index: 442

Text #12

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0742. Min Act: -0.0000

Data Index: 565192 (C4 (Web Text))

Max Activating Token Index: 395

Text #13

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0603. Min Act: -0.0000

Data Index: 1064398 (C4 (Web Text))

Max Activating Token Index: 958

Text #14

Max Range: 1.2897. Min Range: -1.2897

Max Act: 0.9240. Min Act: -0.0000

Data Index: 725820 (C4 (Web Text))

Max Activating Token Index: 760

Text #15

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0049. Min Act: -0.0000

Data Index: 525185 (C4 (Web Text))

Max Activating Token Index: 553

Text #16

Max Range: 1.2897. Min Range: -1.2897

Max Act: 0.9905. Min Act: -0.0000

Data Index: 1306953 (C4 (Web Text))

Max Activating Token Index: 974

Text #17

Max Range: 1.2897. Min Range: -1.2897

Max Act: 0.9835. Min Act: -0.0000

Data Index: 1152203 (C4 (Web Text))

Max Activating Token Index: 817

Text #18

Max Range: 1.2897. Min Range: -1.2897

Max Act: 0.9565. Min Act: -0.0000

Data Index: 104813 (C4 (Web Text))

Max Activating Token Index: 371

Text #19

Max Range: 1.2897. Min Range: -1.2897

Max Act: 1.0130. Min Act: -0.0000

Data Index: 897053 (C4 (Web Text))

Max Activating Token Index: 717